Three dimensional glasses free light field display using eye location

ABSTRACT

In some examples, a three dimensional display system includes a display (for example, a display screen or a display panel), a micro lens array, and an eye tracker to track one or more eyes of a person and to provide eye location information. The display system also includes a rendering processor to render or capture color plus depth images (for example, RGB-D images) or light field images. The display system also includes a light field processor to use the eye location information to convert the rendered color plus depth images or light field images to display images to be provided to the display.

TECHNICAL FIELD

This disclosure relates generally to three dimensional (3D) display.

BACKGROUND

Technology needed to deliver stereo three dimensional (3D) video contentsuch as 3D TV, cinema, gaming, etc. has increasingly entered mass marketproducts (for example, such as Virtual Reality, 3D Cinema, 3Dsmartphones, etc.) Some stereo 3D displays such as active/passiveglasses-based, multi-view lenticular, etc. deliver a different image toeach eye in a stereo pair. These independent images can be, for example,stereoscopically fused in a user's brain, effectively re-creating abinocular experience of 3D visual perception.

In real world situations, when a human is observing a particularsubject, for example, their eyes both converge and focus (oraccommodate) to the distance of the subject. However, in many stereothree dimensional displays, a user is not able to converge and focus (oraccommodate) to the distance of the subject.

BRIEF DESCRIPTION OF THE DRAWINGS

The following detailed description may be better understood byreferencing the accompanying drawings, which contain specific examplesof numerous features of the disclosed subject matter.

FIG. 1 illustrates a real world viewing environment;

FIG. 2 illustrates a three dimensional (3D) display viewing environment;

FIG. 3 illustrates a three dimensional (3D) display system;

FIG. 4 illustrates temporal multiplexing;

FIG. 5 illustrates an eye box;

FIG. 6A illustrates an eye box;

FIG. 6B illustrates a display and a micro lens array;

FIG. 7 illustrates an eye box;

FIG. 8 illustrates a display image processing environment;

FIG. 9 illustrates a computing device;

FIG. 10 illustrates one or more processor and one or more tangible,non-transitory, computer-readable media;

In some cases, the same numbers are used throughout the disclosure andthe figures to reference like components and features. In some cases,numbers in the 100 series refer to features originally found in FIG. 1;numbers in the 200 series refer to features originally found in FIG. 2;and so on.

DESCRIPTION OF THE EMBODIMENTS

Some embodiments relate to three dimensional (3D) display. Someembodiments relate to 3D display with eye tracking (and/or with pupiltracking). Some embodiments relate to a 3D display that solvesconvergence and accommodation issues.

In three dimensional displays, a user may not be able to converge andfocus (or accommodate) to the distance of the subject. This can bereferred to as a vergence-accommodation conflict that can occur in 3Ddisplays, and can also be referred to as a convergence-accommodationconflict. This conflict relates to eye focus and convergence issues, andcan cause eye fatigue, dizziness, nausea, etc., particularly afterextended use. Tensor displays can sometimes solve convergence andaccommodation (focus) problems, but the efficiency of such a display canbe very low. These displays can suffer from diffraction, Moire, etc.,and can require a large amount of computation for optimization purposes.Volumetric displays can be bulky, and often require moving mechanicalparts. Many volumetric displays also do not support occlusion, and cansuffer from reduced visual quality such as being unable to render colorimages.

In some embodiments, a three dimensional display system can beimplemented that is bright, presents full color images, and allows forcorrect occlusion. In some embodiments, a three dimensional display canbe compact but still have a large depth field.

FIG. 1 illustrates an environment 100 (for example, a “real world”environment) including a user 102 (or viewer 102). The user 102 has eyes104. The user's eyes 104 look out toward an object 106 (for example, areal world object 106) along lines of vision illustrated by dotted lines110 in FIG. 1. Dotted line 112 illustrates a focus line ending at afocus point at object 106. Dotted lines 110 end at a convergence pointthat is also at object 106. In some embodiments, environment 100illustrates real three dimensional (3D) vision in physical space. Theconvergent point and the focus point in FIG. 1 are on the same depthplane. That is, the convergent point and the focus point in the realworld environment 100 is the same (for example, at a real world object106).

FIG. 2 illustrates an environment 200 (for example, a three dimensionaldisplay environment) including a user 202 (or viewer 202). The user 202has eyes 204. The user's eyes 204 look out toward an apparent locationof a three dimensional (3D) displayed object 206 displayed on a threedimensional (3D) display screen 208 along lines of vision illustrated bydotted lines 210 in FIG. 2. Dotted line 212 illustrates a focus lineending at a three dimensional (3D) focus point (for example, a focus oraccommodation point 214) at 3D display screen 208. Dotted lines 210 endat a convergence point that is at the apparent location of the displayed3D object 206. In some embodiments, environment 200 illustrates threedimensional (3D) vision by a user 202 viewing an apparent object 206being displayed on a 3D display screen 208. In some embodiments, theeyes 204 of the user 202 are forced to decouple the focus point on the3D display screen 208 from the convergence point at the apparentlocation of the 3D object 206. This is known as vergence-accommodationconflict, convergence-accommodation conflict, and/or a convergence andaccommodation conflict problem, for example.

In real world environments such as, for example, environment 100 of FIG.1, when a human user is observing a particular subject, for example,where the eyes of the user both converge and focus (or accommodate) tothe distance of the subject (for example, real world object 106).However, in many stereo three dimensional display environments such as,for example, environment 200 of FIG. 2, a user is not able to convergeand focus (or accommodate) to the distance of the apparent location ofthe subject (for example, to the apparent location 206 of the threedimensionally displayed object).

In some three dimensional (3D) display systems, the user's eyes mustaccommodate to a single focal plane (for example, the focal plane of 3Ddisplay screen 208) in order to see a scene in focus. In the case of 3Dvideo such as 3D television (TV) or a 3D movie, for example, the focalplane is the physical display screen itself. However, the user's eyesmay converge to one or more apparent locations of one or more 3D objectsbeing displayed. These apparent locations can be in front of and/orbehind the screen. The distance from the user's eyes to the displayscreen and the distance from the user's eyes to the apparent location(s)of the 3D object(s) appearing in front of and/or behind the screen willnot match in many situations. This mismatch of distance from the eyes tothe focus point (or accommodation point) at the display screen and thedistance from the eyes to the 3D convergence point(s) at the apparentlocation(s) of displayed 3D objects can be referred to as theconvergence-accommodation conflict (or the vergence-accommodationconflict, or the convergence and accommodation conflict problem). As aresult of this conflict, the user may experience headaches, fatigue,eyestrain, etc. This problem may cause health implications, particularlyamong children whose visual systems are still developing.

FIG. 3 illustrates a three dimensional (3D) display system viewingenvironment 300. Environment 300 includes a user 302 (or viewer 302) ofa display system. The user 302 has eyes including pupils 304. Thedisplay system viewing environment 300 includes a display 312 to beviewed by the user 302. In some embodiments, display 312 can be any typeof display. In some embodiments, display 312 can be and/or can include adisplay screen and/or a display panel. In some embodiments, display 312can be a display, a display screen, and/or a display panel with anypixel density. In some embodiments, display 312 can be a high pixeldensity display, a high pixel per inch or high PPI display, and/or a 4Kdisplay. In some embodiments, display 312 is a display having a pixeldensity higher than 250 pixels per inch. In some embodiments, display312 is a display with as high of a pixel density as possible. In someembodiments, display 312 can be a mobile phone screen or a tabletscreen, among others. In some embodiments, display 312 can be a liquidcrystal display, an LCD display, a liquid crystal display screen, an LCDdisplay screen, a liquid crystal display panel, and/or an LCD displaypanel, etc.

In some embodiments, a display backlight 314 may be provided behind thedisplay 312. A micro lens array 316 (for example, a long focal lengthmicro lens array and/or a long focal length micro lens array with an Fnumber larger than 8) is provided in front of the display 312 betweenthe display 312 and the eyes of the user 302. In some embodiments, microlens array 316 is a long focal length micro lens array with high focallength. For example, in some embodiments, micro lens array 316 is a longfocal length micro lens array with high focal length with an F numberbetween 8 and 30. In some embodiments, micro lens array 316 has a highfocal length, depending on viewing distance and eye box size (or viewingarea size), with an F number varying between 8 and 30. A pupil trackingdevice 318 can be used to track the pupils 304 of the user 302, and sendany result(s) to a computing device 322 and a light field processingunit 324. In some embodiments, for example, pupil tracking device 318can be a real-time 3D pupil tracking system. Display 312 is used todisplay apparent locations of three dimensional objects 320, which canappear to user 302 to be at the display 312 (for example, at a plane ofthe display 312), in front of the display 312 (between the display 312and the user 302), and/or behind the display 312.

Computing device 322 can include, for example, a graphics engine. Thecomputing device 322 can render color images and/or color plus depthimages (for example, can render any color components such as red, green,blue color components with or without depth, including for example, RGBor RBG-D images) according to the user's left and/or right pupilposition, and send the rendered images to the light field processingunit 324. It is noted that in some embodiments, the images can becaptured rather than rendered by computing device 322. In someembodiments, computing device 322 can capture or render stereo colorplus depth images or light field images. The light field processing unit324 can use pupil location information from the pupil tracking device318 to convert the images received from the computing device 322 tostereo integral imagery using, for example, screen space ray tracing.Although screen space ray tracing is described herein, there are manydifferent ways that the images may be converted. For example, renderingand post processing can be implemented in many different ways accordingto some embodiments. Many post processing techniques other than screenspace ray tracing may be implemented according to some embodiments. Thelight field processing unit 324 can provide a light field image streamto the display screen 312 for display. In some embodiments, thebacklight 314 (for example, a directional backlight) can steer light tocorresponding pupil positions.

In some embodiments, the display system illustrated in FIG. 3 is aflexible light steering apparatus enabling a plurality of applications.Some embodiments implement single-viewer glasses-free three dimensional(3D) display that does not suffer from a convergence and accommodationconflict problem. In some embodiments, such display is compact,lightweight, and computationally and energy efficient relative to otherdisplays such as, for example, brute force integral displays requiringdozens of gigapixels, a tensor display, which may be constrained bydiffraction limitations making generation of sufficient density of viewdifficult in order to support accommodation, or bulky and impracticalvolumetric displays. In some embodiments, a directional backlightapparatus can be implemented for further benefits.

In some embodiments, the light steering properties of display system inFIG. 3 can be used to generate a correct light field for light rays thatpass through the viewer's eye pupils. Generating light field directed toeye pupil location can allow using currently available commoditydisplays to create 3D images with light rays with high enough density toproduce correct accommodation (or focus) and convergence cues.

In some embodiments, extremely high view densities can be achievedwithin a small eye box. The position of the small eye box can be alignedwith the viewer's pupil position (for example, in response to a pupiltracking device such as pupil tracking device 318). The size of thesmall eye box can correspond to a typical human eye pupil size range.High view density can be achieved within a small eye box by placing amicro lens array (MLA) (for example, such as a long focal length microlens array and/or micro lens array 316) in front of a display (forexample, such as a high pixel density display and/or display 312), witha specific space in between the micro lens array and the display. Thelocation of the eye box can be changed by shifting the image to bedisplayed on the display in response to the user tracking apparatus (forexample, in response to a pupil tracking device such as pupil trackingdevice 318).

In some embodiments, a rendering process (for example, a renderingprocess implemented by computing device 322) is dependent on positionsof the viewer's eyes (for example, dependent on positions of pupils 304of user 302 tracked by pupil tracking device 318). In some embodiments,a capturing process (for example, a capturing process implemented bycomputing device 322) is dependent on positions of the viewer's eyes(for example, dependent on positions of pupils 304 of user 302 trackedby pupil tracking device 318). In some embodiments, a real time 3Dtracking system (for example, pupil tracking device 318) can be used totrack a viewer's pupil positions. The tracked pupil positions can beused to position capturing or rendering camera projection centers,and/or to digitally fine-tune eye box alignment.

Some embodiments relate to glasses-free 3D display using an integralimaging system, a 3D pupil tracking device, and a light-field processingunit that can deliver view density high enough for both left and righteyes of a user to overcome vergence-accommodation conflict orconvergence-accommodation conflict (that is, convergence andaccommodation conflict issues).

In some embodiments, a 3D pupil tracking system (such as pupil trackingdevice 318) tracks the pupil position of a viewer of a display, andsends the result to a computer (for example, such as computing device322) and to a light field processing unit (for example, such as lightfield processing unit 324). The computing device captures and/or rendersstereo images (for example, RGB and/or RGB-D images) according to theviewer's left and/or right pupil position. The light field processingunit uses pupil location information to convert the captured and/orrendered images to integral imagery using, for example, screen space raytracing or any other view interpolation/synthesis technique. The imageis then displayed on a high resolution display screen (for example, ondisplay 312). At the same time, in some embodiments, a directionalbacklight (for example, such as backlight 314) steers light tocorresponding pupil positions.

In some embodiments, a micro lens array (MLA) such as, for example,micro lens array 316, is placed in front of a high pixel density displaymodule such as, for example, display 312. The display module and themicro lens array are spaced at the focal length of the micro lens arrayso that the rays from each pixel on the display pass through each microlens on the micro lens array to form a collimated beam. Given thedistance d_(z) between the micro lens array and a user eye viewpoint infront of the micro lens array, a largest eye box occurs when theintegral image pitch size w_(p) is:

$\begin{matrix}{w_{p} = \frac{\left( {d_{z} + f} \right)p_{l}}{d_{z}p_{p}}} & {{EQUATION}\mspace{14mu} 1}\end{matrix}$

Where f is the focal length of the micro lens array, p_(l) is the lenspitch of the micro lens array, and p_(p) is the pixel pitch of thedisplay. The eye box size w_(e) can be calculated using the following:

$\begin{matrix}{w_{e} = \frac{d_{z}w_{p}p_{p}}{f}} & {{EQUATION}\mspace{14mu} 2}\end{matrix}$

When the viewer's eyes are perfectly located in the center of the eyebox, they are able to observe a correct light field image. The rays fromone pixel travel through the micro lens above that pixel, and alsotravel through neighboring micro lenses, forming replica eye boxes. Whenthe viewer's eyes move out of the primary eye box, they will perceive achange in the light field image, and then enter a replica eye box.

In some embodiments illustrated in FIG. 3 and/or other embodiments, ifyou take a display with a certain image pattern displayed thereon, andplace a lens array in front of the display, a user can see a full threedimensional image. This can be accomplished without the user wearingglasses, but merely looking at an area of the display and lens array.Depending on parameters of the display and the lens array, the size ofthe area that the user will see the 3D image may vary. The area wherethe user sees this 3D image may be referred to as a primary viewing zone(viewing zone as used herein is also referred to as an eye box).However, if the user moves outside a primary viewing zone (primary eyebox), the user may also see the same or similar image in one or moresecondary viewing zones (secondary eye boxes) that may repeat throughvarious user viewing areas (eye boxes). There may be a number ofsecondary viewing zones, the number of secondary viewing zones dependingon what the lens array may allow the user to see.

The resolution of the image (and/or 3D image quality), focus capabilityof the image, etc. can depend on an overall number of pixels beamed intoa viewing zone. The smaller the viewing zone, the denser the lightfield. The wider the viewing zone, the sparser the light field. Withsparser light fields, focus will degrade and 3D image quality willgradually become unacceptable

In an implementation in which a lens array is placed in front of adisplay and the viewing zone is made small (for example, slightly largerthan the size of a viewer's pupil), a very dense light field can becreated. However, one pupil may see the image very well, but the otherpupil may not see the image well at all (for example, the other pupilmay see the same image or a corrupted image because it is betweenviewing zones). Additionally, if a user shifts their eye, their eye mayleave one viewing zone and enter another viewing zone. If the viewingzone (eye box) is divided into two portions (for example, one portionfor left eye and another portion for right eye), the left eye may seeone repetition of a left portion of the viewing zone, and the right eyemay see another repetition of a right portion of the viewing zone, thenthe user can view a stereo dense light field to view a 3D image with theuser's two eyes. Another approach is to use one eye box per eye.However, if a regular backlight is used where light goes in multipledirections, all eye boxes (viewing zones) will be lit up with allrepetitions of the viewing zones.

In some embodiments, a directional backlight can be used to emit lightall in a certain direction. In this manner, directional backlightcontrol can be used to light up only one viewing zone at a particulartime, and repetitions of the viewing zone are not visible. For example,in some embodiments, a directional backlight emits light that is onlydirected toward one eye box (or viewing zone) in the area of an eye(and/or a pupil) of a user at one particular time. In some embodiments,pupil position is tracked, so the directional backlight can becontrolled to send light to a particular tracked pupil at a particulartime. In some embodiments, particular light emitting diodes (LEDs) ofthe backlight can be turned on and particular other LEDs of thebacklight can be turned off to direct the controlled light emitted fromthe backlight to emit the light in the area of the viewer's eye (and/orpupil). In this manner, direction of the emitted light of thedirectional backlight can be changed according to eye movement (and/orpupil movement). Such directional backlight can be time multiplexed(also referred to herein as temporal multiplexed) between eyes of aviewer. In some embodiments, eyes (and/or pupils) of more than one usercan be time multiplexed according to some embodiments. In someembodiments, for one particular user, the time multiplexing occurs at afast frequency between eyes of a user (and/or pupils of a user) so thatthe image appears continuous to the user. For example, in someembodiments, the frequency can be 120 Hz for two eyes of a user (60 Hzfor each eye). In some embodiments, the frequency is greater than 120 Hz(greater than 60 Hz for each eye).

In some embodiments, as described above, a high view density light fieldconcentrated around one or more viewer's eye pupils is generated by asystem using a long focal length micro lens array and a directionalbacklight. Such a system directs a light field into a small eye box (forexample, in some embodiments, a small 10 mm by 10 mm eye box) with manyviews (for example, 20 by 20 views). In some embodiments, the eye boxposition is changed using a controllable directional backlight system.In this manner, at any particular moment the directional backlight cansteer light into only one eye. Time multiplexing can be used to deliverrequired light field into both eyes, by changing the eye box positionand displayed content at speeds exceeding a human eye flicker threshold.

FIG. 4 illustrates temporal multiplexing 400 (and/or time multiplexing400). In some embodiments, FIG. 4 illustrates temporal multiplexing(and/or time multiplexing) with directional backlight control. FIG. 4illustrates a user 402 with left and right eyes at a first timeframe(frame 1), a second timeframe (frame 2), a third timeframe (frame 3),and a fourth timeframe (frame 4), for example. In some embodiments, thetimeframes are at a display rate faster than a human can recognize. Insome examples, the timeframes are at a display rate of 120 Hz, forexample.

FIG. 4 also illustrates a display 412 (for example, a display screenand/or display panel), a backlight 414 (for example, a directionalbacklight that can be directionally controlled), and a micro lens array416. In some embodiments, display 412, backlight 414, and/or micro lensarray 416 can be similar to and/or the same as display 312, backlight314, and/or micro lens array 316 of FIG. 3. At frame 1, a high viewdensity light field 432L is concentrated from backlight 414, display 412and micro lens array 416 to a left eye (and/or a left pupil) of user402. At frame 2, a high view density light field 432R is concentratedfrom backlight 414, display 412 and micro lens array 416 to a right eye(and/or a right pupil) of user 402. At frame 3, a high view densitylight field 432L is concentrated from backlight 414, display 412 andmicro lens array 416 to the left eye (and/or the left pupil) of user402. At frame 4, a high view density light field 432R is concentratedfrom backlight 414, display 412 and micro lens array 416 to the righteye (and/or the right pupil) of user 402. In this manner, displaybacklight 414 can be a directional backlight that can steer lightalternating between the left and right eye of the user 402. Thissteering of light can be based, for example, on tracked eye locationinformation (for example, tracked pupil location information). Thetracked eye location information can be tracked eye location informationfrom an eye tracking device such as, for example, pupil tacking device318. In some embodiments, the display backlight 414 can steer lightalternating between the left and right eye based on the tracked eyelocation information (and/or based on the tracked pupil locationinformation) at a refresh rate that is higher than a human perceivablerefresh rate (for example, at 120 Hz). For example, the frames in FIG. 4can be alternated at a refresh rate that is higher than a humanperceivable refresh rate (such as a refresh rate of 120 Hz).

In some embodiments, a high view density light field (for example, highview density light field 432L and/or high view density light field 432R)can be concentrated around one or more viewer's eye pupils (for example,can be concentrated around eyes and/or eye pupils of viewer 402). Thehigh view light field(s) can be generated by a system using a long focallength micro lens array (for example, micro lens array 416) and adirectional backlight (for example, backlight 414). Such a systemdirects a light field (for example, light field 432L and/or light field432R) into a small eye box (for example, in some embodiments, a small 10mm by 10 mm eye box) with many views (for example, 20 by 20 views). Insome embodiments, the eye box position is changed using a controllabledirectional backlight system (for example, including backlight 414). Inthis manner, at any particular moment the directional backlight cansteer light into only one eye as illustrated, for example, in FIG. 4.Time multiplexing can be used to deliver required light field into botheyes, by changing the eye box position and displayed content at speedsexceeding a human eye flicker threshold (for example, at 120 Hz).

FIG. 4 has been described herein as time multiplexing between two eyesof a single user. However, in some embodiments, the time multiplexingillustrated in FIG. 4 or described herein is not limited to a singleuser. In some embodiments, time multiplexing of directional backlightcan be implemented between more than one user (for example, between eyesof multiple users at a high enough refresh rate so that it is notapparent to any one user). Steering of directional backlight is notlimited to one user. If display frequency is high enough, two or moreusers can be addressed simultaneously. For example, in some embodiments,in frame 1 a first eye of a first user receives directional backlight,in frame 2 a first eye of a second user receives directional backlight,in frame 3 a second eye of the first user receives directionalbacklight, and in frame 4 a second eye of the second user receivesdirectional backlight. In other embodiments, other numbers of users andorder of eyes receiving directional backlight at particular times can beimplemented. In some embodiments, a sequential system is implemented.For example, some embodiments relate to a time multiplexed system inwhich light can be steered (for example, from a directional backlight)to individual positions or spaces near a display in a time multiplexedmanner.

In some embodiments, with controlled backlight or without controlledbacklight, the integral image displayed on the display can define thesize and position of the eye box. By shifting the integral imagedisplayed on the display, the eye box center can be shifted to alignwith either the left or the right pupil position.

FIG. 5 illustrates an eye box 500 that is subdivided into left (L) andright (R) parts. Eye box 500 includes left (L) and right (R) center eyebox 502. The other eye boxes 500 other than center eye boxes 502 arerepetitions of the center eye box 502. In some embodiments, eye box 500is referred to as a divided eye box or a subdivided eye box, since it isdivided into left (L) and right (R) portions. In some embodiments, eyebox 500 can be used in conjunction with the system of FIG. 3 and/or ofFIG. 4 in order to deliver a required light field into the eyes (forexample, into the pupils) of one or more user.

FIG. 6A illustrates an eye box 600A from a front view. FIG. 6Billustrates a combination of a display 612B and a micro lens array 616Bfrom a side view. In some embodiments, FIG. 6A illustrates good viewingarea 602A and bad viewing areas 604A from a front view. In someembodiments, the side view 600B is a view from the side of front view600A. In some embodiments, micro lens array 616B is a long focal lengthmicro lens array. In some embodiments, display 612B is similar to and/orthe same as display 312 of FIG. 3. In some embodiments, micro lens array616B is similar to and/or the same as micro lens array 316 of FIG. 3. Insome embodiments, eye box 600A can be used in conjunction with thesystem of FIG. 3 and/or of FIG. 4 in order to deliver a required lightfield into the eyes (for example, into the pupils) of one or more user.

In some embodiments, a high view density light field is generated in asmall eye box using a long focal length micro lens array (for example,micro lens array 616B) and a divided eye box (for example, divided eyebox 500 and/or divided eye box 600A).

In some embodiments, by dividing the eye box into left (L) and right (R)parts, the eye box can be arranged in a manner such that the left partof the eye box will cover the left eye, and a repetition of the rightpart of the eye box will cover the right eye. In this manner, the viewerwill see a correct light field when at a correct viewing distance. Insome embodiments, when a user's IPD is close to an integral multiple ofthe eye box size, the user will not be able to perceive a correct lightfield image since both eyes will see the same sub eye box (for exampleboth eyes will see a right eye box or both eyes will see a left eyebox). In some embodiments, this problem can be solved using micro lensarray optics that can change focal lengths and spacing between microlens array and screen, or by changing the viewing distance whichdetermines the eye box size.

In some embodiments, a compact and controllable directional backlightsystem (for example such as backlight 314 of FIG. 3 and/or backlight 414of FIG. 4) is used to overcome limitations of a divided eye box system.For example, a divided eye box system can produce a correct light fieldat certain fixed viewing distances when an IPD of a viewer (such as ahorizontal IPD of a viewer) is an odd multiple of half the eye boxwidth. As illustrated in FIG. 3, for example, when IPD=5*W_(e)/2, an Lportion of the eye box lines up on the left eye of the viewer and an Rportion of the eye box lines up on the right eye of the viewer. However,adjustment and/or use of a backlight may be necessary for a viewer withIPD=6*W_(e)/2, where an L portion of the eye box lines up on the leftand an L portion of the eye box also lines up on the right, for example.Similar arrangements can occur with eye box 600A illustrated in FIG. 6A.

In some embodiments, the eye box shape is not required to be square. Insome embodiments, however, the eye box is in a shape that can be tiledin a uniform grip. If the viewer's head is rotated with respect to thedisplay screen, the viewer's left and right eyes may have differentapparent heights relative to the display. Therefore, in someembodiments, left and right portions of the eye box can be moved up ordown.

FIG. 7 illustrates an eye box 700. In some embodiments, eye box 700includes first eye box portions 702 (for example, left eye box parts702) and second eye box portions 704 (for example, right eye box parts704). As discussed above, if the viewer's head is rotated with respectto the display screen, for example, the viewer's left and right eyes mayhave different apparent heights relative to the display. Therefore, asillustrated in FIG. 7, portions 702 (for example, left portions 702) ofthe eye box 700 can be moved up or down, and/or portions 704 (forexample, right portions 704) can be moved up or down so that portions702 and 604 are at different heights. In some embodiments, eye box 700can be used in conjunction with the system of FIG. 3 and/or of FIG. 4 inorder to deliver a required light field into the eyes (for example, intothe pupils) of one or more user.

Some portions of eye boxes have been illustrated herein as beingrectangular in shape. For example, L and R eye box portions in FIGS. 5and 7 are illustrated as being rectangular. However, in someembodiments, this may cause less than half of the light rays to beutilized since a rectangular shape does not match up well with the shapeof a pupil. However, in some embodiments, a micro lens array can bedesigned with a different micro lens aspect ratio. For example, in someembodiments, a micro lens array design can be used with a micro lensaspect ratio equal to 2, resulting, for example, in an arrangement whereeach subdivided (or divided) eye box can be of a square shape.

In some embodiments, in order to generate an image to be displayed onthe 3D display, capturing and/or rendering based on the pupil positionof a user (viewer) can be performed by a computing device (for example,using graphics hardware) to generate intermedia data that encapsulatethe geometry and texture information of the scene (such as, for example,RGB-D images). The data (such as RGB-D images) is then transmitted to alight field processing unit. The light field processing unit uses thereal-time pupil position to calculate an optimal eye box size and animage offset needed to align the eye box center with pupil position.Then the light field processing unit can convert the image (such as theRGB-D image) to the final integral image. This can be implemented, forexample, using screen-space ray tracing, or according to othertechniques. In some embodiments, instead of using a graphic engine togenerate the RGB-D images, captured RGB-D images can be transmitted to alight field processing unit to generate an image to be displayed on the3D display.

Screen-space ray tracing is a very efficient post-processing techniquefor generating approximations of reflection, refraction, glossyreflection, ambient occlusion, and/or global illumination. This can beimplemented at a much lower cost than some ray tracing techniques. Insome embodiments, screen-space ray tracing is used to generate lightfield renderings from RGB-D data. In some embodiments, techniques otherthan screen-space ray tracing may be used (for example, according tosome embodiments, any post processing technique may be used).

FIG. 8 illustrates display image processing in a three dimensional (3D)display environment 800. Environment 800 includes a display 812 (and/ora display screen 812, and/or a display panel 812) and a micro lens array816. Display 812 can display three dimensional images to be viewed by auser or viewer. In some embodiments, display 812 can be the same as orsimilar to display 312, for example. In some embodiments, display 812can be any type of display, any type of display screen, and/or any typeof display panel. In some embodiments, display 812 can be a mobile phonescreen or a tablet screen, among others. In some embodiments, a displaybacklight (not shown in FIG. 8) may be provided behind the display 812.Micro lens array 816 can be a long focal length micro lens array, and isprovided in front of the display 812 between the display 812 and theeyes of a user. Display 812 is used to display apparent locations ofthree dimensional (3D) objects 820, which can appear to a user to be atthe display 812 (for example, at a plane of the display 812, in front ofthe display 812 (between the display 812 and the user, and/or behind thedisplay 812. In FIG. 8, the objects 820 are displayed at apparentlocations behind the display 812 (above the display 812 in FIG. 8). Acapturing and/or rendering camera 832 is used to capture and/or renderimages through a near clip plane 842, micro lens array 816, display 812,and a far clip plane 844.

In some embodiments, a scene including objects 820 has been capturedand/or rendered to a canvas at the near clip plane 842. Since the scenehas been captured and/or rendered, color and depth at any point isavailable. For a pixel P on the display 812, the location of the pixel Pand the optical center of the lenslet (or lens within the micro lensarray) in front of the pixel P defines a ray R in space. The color ofpixel P can be defined by the color of the intersecting point of ray Rwith the three dimensional (3D) scene (for example, at point C). Asimple one dimensional (1D) search on the canvas of the near clip plane842 can be used to find the intersecting point C using the followingsteps:

1. Compute the intersection of ray R with both the near clip plane 842and the far clip plane 844. In the case of FIG. 8, these two points arepoint A at the near clip plane 842 and point B at the far clip plane844.

2. Project point B onto the near clip plane 842 to get point B′. Thisprojection can be performed by drawing a line between point B and theviewpoint of the capturing and/or rendering camera 832.

3. Interpolate from point A to point B′. This interpolation can occur inthe two dimensional (2D) plane represented by near clip plane 842. Eachpixel in the sequence can be efficiently computed using the Bresenhaminterpolation algorithm (or Bresenham's line algorithm, or digitaldifferential analyzer algorithm, or DDA line algorithm), for example. Insome embodiments, Bresenham's line algorithm (or other algorithm) can beused to determine points of an n-dimensional raster that should beselected in order to form a close approximation to a straight linebetween two points. In some embodiments, any of the Bresenham family ofalgorithms extending or modifying Bresenham's original algorithm may beused. See, for example,https://en.wikipedia.org/wiki/Bresenham%27s_line_algorithm.

4. For each point C′ generated by the interpolation, the depth is readfrom a precomputed two dimensional (2D) canvas. The depth of thecorresponding 3D point C on ray R is also computed. Since the 2D linesegment from point A to point B′ is a 2D projection of the 3D line frompoint A to point B, for any point C′ a corresponding point C can bedirectly computed. Points C′ are repeatedly generated and tested untilthe depth of the 3D point C is larger than the depth read from theprecomputed 2D canvas, or the 3D point is outside the frustrum (that is,past the far clipping plane 844). That test can be used to determinewhether the correct point C in the virtual scene has been found, or ifthere is no virtual object associated with that pixel. If there is novirtual object associated with that pixel, then the color of C can beset to a background color.

5. Set the color of the pixel P to the color of pixel C and stop theprocess.

In some embodiments, techniques described herein can be used to build acomputational display that is practical for users. In some embodiments,computational displays can be used to display 3D content without causingeyestrain or requiring 3D glasses. In some embodiments, displaysdescribed and/or illustrated herein can be included in all form factors(for example, all displays including wearables, phones, tablets,laptops, desktops, and/or far-eye displays).

In some embodiments, the image to be displayed on the 3D display can berendered directly with ray tracing, and/or with any other techniques.With tracked pupil location and viewing distance, the eye box size andposition can be computed using Equation 1 and Equation 2. In someembodiments, each single pixel on the display will only be visible fromone micro lens in any given eye box. Tracing the ray that passes thoughthe pixel center and the mirco lens optical center in the virtual scenewill return the pixel color for that particular pixel in the finalimage.

In some embodiments, the image to be displayed on the 3D display can berendered directly with conventional rasterization using multi frustums.The multi frustums for the rendering cameras are defined by the eye boxsize, eye box position, and screen size. The number of pixels visiblefrom one single lens defines the number of rendering cameras needed. Incase the number of pixels visible from one single micro lens is not anintegral number, the rendering camera number can be up sampled to alarger integer number. For example, the number of pixels under one microlens is 10.5*10.5, but it is impossible to have 10.5*10.5 renderingcameras. As an alternative, 15*15 rending cameras can be used. Thefrustums are defined by the projection center of the frustums, which arethe uniform 15*15 2D grid sample of the eye box, and the four corners ofthe display. The resolution need for each camera is defined by thenumber of micro lens on the display. The final integral image can begenerated by interleaving the 15*15 rendered image in reverse directionand then down sample by the inverse of the up sample ratio (10.5/15).Depending on the location of the pupil, the integral image will need tobe shifted.

In some embodiments, directional backlight based time multiplexing isimplemented. In some embodiments, an eye box (also referred to herein asa viewing area) is split (for example, between left and right eyes). Insome embodiments, various content generation approaches may beimplemented. In some embodiments, any rendering implementation may beused. In some embodiments, screen space ray tracing may be used (forexample, in some embodiments, screen space ray tracing on color plusdepth images such as RGB-D images). In some embodiments, ray tracing maybe implemented. In some embodiments, captured data synthesis may beimplemented (for example, captured image data synthesis such as capturedRGB image data or captured RGB-D image data).

FIG. 9 is a block diagram of an example of a computing device 900. Insome embodiments, computing device 900 can be included in a device thatcan implement viewing mode adjustment. In some embodiments, computingdevice 900 can implement pupil (or eye) tracking, image rendering, imageprocessing, etc. as described in this specification and/or illustratedin the drawings. In some embodiments, computing device 900 can beincluded as a portion or all of a pupil tracking device (for example,pupil tracking device 318). In some embodiments, computing device 900can be included as a portion or all of a computing device implementingimage rendering (for example, computing device 322). In someembodiments, computing device 900 can be included as a portion or all ofa light field processing unit (for example, light field processing unit324). In some embodiments, the computing device 900 may be included in,for example, a display system, among others. The computing device 900may include a processor 902 that is adapted to execute storedinstructions, as well as a memory device 904 (and/or storage device 904)that stores instructions that are executable by the processor 902. Theprocessor 902 can be a single core processor, a multi-core processor, acomputing cluster, or any number of other configurations. The memorydevice 904 can be a memory device and/or a storage device, and caninclude volatile storage, non-volatile storage, random access memory,read only memory, flash memory, and/or any other suitable memory and/orstorage systems. The instructions that are executed by the processor 902may also be used to implement viewing mode adjustment as described inthis specification.

The processor 902 may also be linked through a system interconnect 906(e.g., PCI®, PCI-Express®, NuBus, etc.) to a display interface 908adapted to connect the computing device 900 to a display device 910. Thedisplay device 910 may include a display screen that is a built-incomponent of the computing device 900. The display device 910 mayinclude a display, a micro lens array, and/or a display backlight, forexample.

In some embodiments, the display interface 908 can include any suitablegraphics processing unit, transmitter, port, physical interconnect, andthe like. In some examples, the display interface 908 can implement anysuitable protocol for transmitting data to the display device 910. Forexample, the display interface 908 can transmit data using ahigh-definition multimedia interface (HDMI) protocol, a DisplayPortprotocol, or some other protocol or communication link, and the like

In some embodiments, display device 910 includes a display controller.In some embodiments, a display controller can provide control signalswithin and/or to the display device. In some embodiments, a displaycontroller can be included in the display interface 908 (and/or insteadof the display interface 908). In some embodiments, a display controllercan be coupled between the display interface 908 and the display device910. In some embodiments, the display controller can be coupled betweenthe display interface 908 and the interconnect 906. In some embodiments,the display controller can be included in the processor 902. In someembodiments, the display controller can implement control of a displayand/or a backlight of display device 910 according to any of theexamples illustrated in any of the drawings and/or as described anywhereherein.

In some embodiments, any of the techniques described in thisspecification can be implemented entirely or partially within thedisplay device 910. In some embodiments, any of the techniques describedin this specification can be implemented entirely or partially within adisplay controller. In some embodiments, any of the techniques describedin this specification can be implemented entirely or partially withinthe processor 902.

In addition, a network interface controller (also referred to herein asa NIC) 912 may be adapted to connect the computing device 900 throughthe system interconnect 906 to a network (not depicted). The network(not depicted) may be a wireless network, a wired network, cellularnetwork, a radio network, a wide area network (WAN), a local areanetwork (LAN), a global position satellite (GPS) network, and/or theInternet, among others.

The processor 902 may be connected through system interconnect 906 to anI/O interface 914. I/O interface 914 can be used to couple interconnect906 with one or more I/O devices 916. One or more input/output (I/O)device interfaces 914 may be adapted to connect the computing hostdevice 900 to one or more I/O devices 916. The I/O devices 916 mayinclude, for example, a keyboard and/or a pointing device, where thepointing device may include a touchpad or a touchscreen, among others.The I/O devices 916 may be built-in components of the computing device900, or may be devices that are externally connected to the computingdevice 900.

In some embodiments, the processor 902 may also be linked through thesystem interconnect 906 to a storage device 918 that can include a harddrive, a solid state drive (SSD), a magnetic drive, an optical drive, aportable drive, a flash drive, a Universal Serial Bus (USB) flash drive,an array of drives, and/or any other type of storage, includingcombinations thereof. In some embodiments, the storage device 918 caninclude any suitable applications. In some embodiments, the storagedevice 918 can include eye tracking (and/or pupil tracking) 920, imagerendering 922, image processing 924, and/or temporal multiplexing 926(such as, for example, temporal multiplexing with directionalbacklight). In some embodiments, eye tracking (and/or pupil tracking)920, image rendering 922, image processing 924, and/or temporalmultiplexing 926 can include instructions that can be executed (forexample, can be executed by processor 902) to perform functionality asdescribed and/or illustrated anywhere in this specification.

It is to be understood that the block diagram of FIG. 9 is not intendedto indicate that the computing device 900 is to include all of thecomponents shown in FIG. 9. Rather, the computing device 900 can includefewer and/or additional components not illustrated in FIG. 9 (e.g.,additional memory components, embedded controllers, additional modules,additional network interfaces, etc.). Furthermore, any of thefunctionalities of eye tracking 920, image rendering 922, imageprocessing 924, and/or temporal multiplexing 926 may be partially, orentirely, implemented in hardware and/or in the processor 902. Forexample, the functionality may be implemented with an applicationspecific integrated circuit, logic implemented in an embeddedcontroller, or in logic implemented in the processor 902, among others.In some embodiments, the functionalities of the eye tracking 920, imagerendering 922, image processing 924, and/or temporal multiplexing 926can be implemented with logic, wherein the logic, as referred to herein,can include any suitable hardware (e.g., a processor, among others),software (e.g., an application, among others), firmware, or any suitablecombination of hardware, software, and firmware.

FIG. 10 is a block diagram of an example of one or more processor andone or more tangible, non-transitory computer readable media. The one ormore tangible, non-transitory, computer-readable media 1000 may beaccessed by a processor 1002 over a computer interconnect 1004.Furthermore, the one or more tangible, non-transitory, computer-readablemedia 1000 may include code to direct the processor 1002 to performoperations as described herein. For example, in some embodiments,computer-readable media 1000 may include code to direct the processor toperform one or more of eye tracking 1006 (and/or pupil tracking 1006),image rendering 1008, image processing 1010, and/or temporalmultiplexing (for example, temporal multiplexing with backlight control)according to some embodiments. In some embodiments, processor 1002 isone or more processors. In some embodiments, processor 1002 can performsimilarly to (and/or the same as) processor 902 of FIG. 9, and/or canperform some or all of the same functions as can be performed byprocessor 902.

Various components discussed in this specification may be implementedusing software components. These software components may be stored onthe one or more tangible, non-transitory, computer-readable media 1000,as indicated in FIG. 10. For example, software components including, forexample, computer readable instructions implementing eye tracking 1006,image rendering 1008, image processing 1010, and/or temporalmultiplexing 1012 (for example, temporal multiplexing with backlightcontrol) may be included in one or more computer readable media 1000according to some embodiments. Eye tracking 1006, image rendering 1008,image processing 1010, and/or temporal multiplexing 1012 may be adaptedto direct the processor 1002 to perform one or more of any of theoperations described in this specification and/or in reference to thedrawings.

It is to be understood that any suitable number of the softwarecomponents shown in FIG. 10 may be included within the one or moretangible, non-transitory computer-readable media 1000. Furthermore, anynumber of additional software components not shown in FIG. 10 may beincluded within the one or more tangible, non-transitory,computer-readable media 1000, depending on the specific application.

Embodiments have been described herein as relating to RGB and/or RGB-Dimages. However, embodiments can relate more generally to any colorimages including RGB images or other color images, and/or can relate tocolor plus depth images including RGB-D images or other color plus depthimages.

Reference in the specification to “one embodiment” or “an embodiment” or“some embodiments” of the disclosed subject matter means that aparticular feature, structure, or characteristic described in connectionwith the embodiment is included in at least one embodiment of thedisclosed subject matter. Thus, the phrase “in one embodiment” or “insome embodiments” may appear in various places throughout thespecification, but the phrase may not necessarily refer to the sameembodiment or embodiments.

Example 1 is a three dimensional display apparatus. The threedimensional display apparatus includes a display (for example, adisplay, a display screen, and/or a display panel) and a micro lensarray. The three dimensional display apparatus also includes an eyetracker to track a plurality of eyes, and to provide eye locationinformation corresponding to the tracking. A rendering processor it torender or capture color plus depth images or light field images. A lightfield processor is to use the eye location information to convert therendered or captured color plus depth images or light field images todisplay images to be provided to the display.

Example 2 includes the display of example 1, including or excludingoptional features. In this example, the rendering processor is to renderlight field images based on the eye location information, and the lightfield processor is to synthesize rendered or captured color, light fieldor multi view images to the required light field image for the display.

Example 3 includes the display of any of examples 1 or 2, including orexcluding optional features. In this example, a display backlight is tosteer backlight based on the eye location information.

Example 4 includes the display of any of examples 1-3, including orexcluding optional features. In this example, the display is a highpixel density display having a pixel density higher than 250 pixels perinch.

Example 5 includes the display of any of examples 1-4, including orexcluding optional features. In this example, the micro lens array is along focal length micro lens array with an F number larger than 8.

Example 6 includes the display of any of examples 1-5, including orexcluding optional features. In this example, the eye tracker is a pupiltracker to track one or more pupils, and the eye location information ispupil location information.

Example 7 includes the display of any of examples 1-6, including orexcluding optional features. In this example, the eye tracker is a threedimensional eye tracker and the eye location information is threedimensional eye location information.

Example 8 includes the display of any of examples 1-7, including orexcluding optional features. In this example, the rendering processor isa graphics engine.

Example 9 includes the display of any of examples 1-8, including orexcluding optional features. In this example, the light field processoris to convert the rendered or captured color plus depth images or lightfield images to stereo integral images to be provided to the display.

Example 10 includes the display of any of examples 1-9, including orexcluding optional features. In this example, the rendering processor isto render each color plus depth image or light field image according tothe tracked location of one or more of the eyes.

Example 11 includes the display of any of examples 1-10, including orexcluding optional features. In this example, the rendering processor isto render the color plus depth images or light field images based on theeye location information.

Example 12 includes the display of any of examples 1-11, including orexcluding optional features. In this example, the light field processoris to use the eye location information to calculate an optimal eye boxsize and to offset displayed images to align a center of an eye box witheye location of the person.

Example 13 includes the display of any of examples 1-12, including orexcluding optional features. In this example, the light field processoris to convert the rendered or captured color plus depth images or lightfield images to display images using one or more post-processingtechnique.

Example 14 includes the display of example 13, including or excludingoptional features. In this example, the one or more post processingtechnique includes screen-space ray tracing.

Example 15 includes the display of any of examples 1-14, including orexcluding optional features. In this example, the light field processoris to provide display images to the display.

Example 16 includes the display of any of examples 1-15, including orexcluding optional features. In this example, a display backlight is tosteer light alternating between a left and right eye based on thetracked eye location information at a refresh rate that is higher than ahuman perceivable refresh rate.

Example 17 is a three dimensional display light field image method. Themethod includes tracking one or more eyes viewing a micro lens array anda display screen to provide eye location information, rendering orcapturing color plus depth images or light field images, and using theeye location information to convert the rendered or captured color plusdepth images or light field images to display images to be provided tothe display.

Example 18 includes the method of example 17, including or excludingoptional features. In this example, light field images are renderedbased on the eye location information, captured color, light field ormulti view images are synthesized to the required light field image forthe display.

Example 19 includes the method of any of examples 17 or 18, including orexcluding optional features. In this example, backlight is steered basedon the eye location information.

Example 20 includes the method of any of examples 17-19, including orexcluding optional features. In this example, the display is a highpixel density display having a pixel density higher than 250 pixels perinch.

Example 21 includes the method of any of examples 17-20, including orexcluding optional features. In this example, the micro lens array is along focal length micro lens array with an F number greater than 8.

Example 22 includes the method of any of examples 17-21, including orexcluding optional features. In this example, one or more pupils viewingthe micro lens array and the display screen are tracked to provide pupillocation information, and the eye location information is pupil locationinformation.

Example 23 includes the method of any of examples 17-22, including orexcluding optional features. In this example, the one or more eyes aretracked in three dimensions, and the eye location information is threedimensional eye location information.

Example 24 includes the method of any of examples 17-23, including orexcluding optional features. In this example, the rendering includesgraphics rendering.

Example 25 includes the method of any of examples 17-24, including orexcluding optional features. In this example, the rendered or capturedcolor plus depth images or light field images are converted to integralimages to be provided to the display.

Example 26 includes the method of any of examples 17-25, including orexcluding optional features. In this example, each color plus depthimage is rendered according to the tracked location of one of the eyes.

Example 27 includes the method of any of examples 17-26, including orexcluding optional features. In this example, the color plus depthimages are rendered based on the eye location information.

Example 28 includes the method of any of examples 17-27, including orexcluding optional features. In this example, the eye locationinformation is used to calculate an optimal eye box size and to offsetdisplayed images to align a center of an eye box with eye location.

Example 29 includes the method of any of examples 17-28, including orexcluding optional features. In this example, the rendered color plusdepth images or captured color plus depth images are converted todisplay images using one or more post processing technique.

Example 30 includes the method of example 29, including or excludingoptional features. In this example, the one or more post processingtechnique includes screen-space ray tracing.

Example 31 includes the method of any of examples 17-30, including orexcluding optional features. In this example, the display images areprovided to the display.

Example 32 includes the method of any of examples 17-31, including orexcluding optional features. In this example, light alternating betweena left and right eye is steered based on the tracked eye locationinformation at a refresh rate that is higher than a human perceivablerefresh rate.

Example 33 is one or more tangible, non-transitory machine readablemedia. The media include a plurality of instructions that, in responseto being executed on at least one processor, cause the at least oneprocessor to track one or more eyes viewing a micro lens array and adisplay screen to provide eye location information, render or capturecolor plus depth images or light field images, and use the eye locationinformation to convert the rendered or captured color plus depth imagesor light field images to display images to be provided to the display.

Example 34 includes the one or more tangible, non-transitory machinereadable media of example 33, including or excluding optional features.In this example, the one or more tangible, non-transitory machinereadable media include a plurality of instructions that, in response tobeing executed on at least one processor, cause the at least oneprocessor to render light field images based on the eye locationinformation, and to synthesize captured color, light field or multi viewimages to the required light field image for the display.

Example 35 includes the one or more tangible, non-transitory machinereadable media of any of examples 33 or 34, including or excludingoptional features. In this example, the one or more tangible,non-transitory machine readable media include a plurality ofinstructions that, in response to being executed on at least oneprocessor, cause the at least one processor to steer backlight based onthe eye location information.

Example 36 includes the one or more tangible, non-transitory machinereadable media of any of examples 33-35, including or excluding optionalfeatures. In this example, the display is a high pixel density displayhaving a pixel density higher than 250 pixels per inch.

Example 37 includes the one or more tangible, non-transitory machinereadable media of any of examples 33-36, including or excluding optionalfeatures. In this example, the micro lens array is a long focal lengthmicro lens array with an F number greater than 8.

Example 38 includes the one or more tangible, non-transitory machinereadable media of any of examples 33-37, including or excluding optionalfeatures. In this example, the one or more tangible, non-transitorymachine readable media include a plurality of instructions that, inresponse to being executed on at least one processor, cause the at leastone processor to track one or more pupils viewing the micro lens arrayand the display screen to provide pupil location information, where theeye location information is pupil location information.

Example 39 includes the one or more tangible, non-transitory machinereadable media of any of examples 33-38, including or excluding optionalfeatures. In this example, the one or more tangible, non-transitorymachine readable media include a plurality of instructions that, inresponse to being executed on at least one processor, cause the at leastone processor to track the one or more eyes in three dimensions, whereinthe eye location information is three dimensional eye locationinformation.

Example 40 includes the one or more tangible, non-transitory machinereadable media of any of examples 33-39, including or excluding optionalfeatures. In this example, the rendering includes graphics rendering.

Example 41 includes the one or more tangible, non-transitory machinereadable media of any of examples 33-40, including or excluding optionalfeatures. In this example, the one or more tangible, non-transitorymachine readable media include a plurality of instructions that, inresponse to being executed on at least one processor, cause the at leastone processor to convert the rendered or captured color plus depthimages or light field images to integral images to be provided to thedisplay.

Example 42 includes the one or more tangible, non-transitory machinereadable media of any of examples 33-41, including or excluding optionalfeatures. In this example, the one or more tangible, non-transitorymachine readable media include a plurality of instructions that, inresponse to being executed on at least one processor, cause the at leastone processor to render each color plus depth image according to thetracked location of one of the eyes.

Example 43 includes the one or more tangible, non-transitory machinereadable media of any of examples 33-42, including or excluding optionalfeatures. In this example, the one or more tangible, non-transitorymachine readable media include a plurality of instructions that, inresponse to being executed on at least one processor, cause the at leastone processor to render the color plus depth images based on the eyelocation information.

Example 44 includes the one or more tangible, non-transitory machinereadable media of any of examples 33-43, including or excluding optionalfeatures. In this example, the one or more tangible, non-transitorymachine readable media include a plurality of instructions that, inresponse to being executed on at least one processor, cause the at leastone processor to calculate an optimal eye box size and to offsetdisplayed images to align a center of an eye box with eye location.

Example 45 includes the one or more tangible, non-transitory machinereadable media of any of examples 33-44, including or excluding optionalfeatures. In this example, the one or more tangible, non-transitorymachine readable media include a plurality of instructions that, inresponse to being executed on at least one processor, cause the at leastone processor to convert the rendered color plus depth images orcaptured color plus depth images to display images using one or morepost processing technique.

Example 46 includes the one or more tangible, non-transitory machinereadable media of any of example 45, including or excluding optionalfeatures. In this example, the one or more post processing techniqueincludes screen-space ray tracing.

Example 47 includes the one or more tangible, non-transitory machinereadable media of any of examples 33-46, including or excluding optionalfeatures. In this example, the one or more tangible, non-transitorymachine readable media include a plurality of instructions that, inresponse to being executed on at least one processor, cause the at leastone processor to provide the display images to the display.

Example 48 includes the one or more tangible, non-transitory machinereadable media of any of examples 33-47, including or excluding optionalfeatures. In this example, the one or more tangible, non-transitorymachine readable media include a plurality of instructions that, inresponse to being executed on at least one processor, cause the at leastone processor to steer light alternating between a left and right eyebased on the tracked eye location information at a refresh rate that ishigher than a human perceivable refresh rate.

Example 49 is a three dimensional display apparatus including a display,a micro lens array, means for tracking a plurality of eyes, means forproviding eye location information corresponding to the tracking, meansfor rendering or capturing color plus depth images or light fieldimages, and means for using the eye location information to convert therendered or captured color plus depth images or light field images todisplay images to be provided to the display.

Example 50 includes the three dimensional display apparatus of example49, including or excluding optional features. In this example, theapparatus includes means for rendering light field images based on theeye location information, and means for synthesizing rendered orcaptured color, light field or multi view images to the required lightfield image for the display.

Example 51 includes the three dimensional display apparatus of any ofexamples 49 or 50, including or excluding optional features. In thisexample, the apparatus includes means for steering backlight based onthe eye location information.

Example 52 includes the three dimensional display apparatus of any ofexamples 49-51, including or excluding optional features. In thisexample, the display is a high pixel density display having a pixeldensity higher than 250 pixels per inch.

Example 53 includes the three dimensional display apparatus of any ofexamples 49-52, including or excluding optional features. In thisexample, the micro lens array is a long focal length micro lens arraywith an F number larger than 8.

Example 54 includes the three dimensional display apparatus of any ofexamples 49-53, including or excluding optional features. In thisexample, the apparatus includes means for tracking one or more pupils,wherein the eye location information is pupil location information.

Example 55 includes the three dimensional display apparatus of any ofexamples 49-54, including or excluding optional features. In thisexample, the apparatus includes means for tracking one or more eyes inthree dimensions, where the eye location information is threedimensional eye location information.

Example 56 includes the three dimensional display apparatus of any ofexamples 49-55, including or excluding optional features. In thisexample, the means for rendering comprises a means for renderinggraphics.

Example 57 includes the three dimensional display apparatus of any ofexamples 49-56, including or excluding optional features. In thisexample, the apparatus includes means for converting the rendered orcaptured color plus depth images or light field images to stereointegral images to be provided to the display.

Example 58 includes the three dimensional display apparatus of any ofexamples 49-57, including or excluding optional features. In thisexample, the apparatus includes means for rendering each color plusdepth image or light field image according to the tracked location ofone of the eyes.

Example 59 includes the three dimensional display apparatus of any ofexamples 49-58, including or excluding optional features. In thisexample, the apparatus includes means for rendering the color plus depthimages or light field images based on the eye location information.

Example 60 includes the three dimensional display apparatus of any ofexamples 49-59, including or excluding optional features. In thisexample, the apparatus includes means for using the eye locationinformation to calculate an optimal eye box size and to offset displayedimages to align a center of an eye box with eye location of the person.

Example 61 includes the three dimensional display apparatus of any ofexamples 49-60, including or excluding optional features. In thisexample, the apparatus includes means for converting the rendered orcaptured color plus depth images or light field images to display imagesusing one or more post-processing technique.

Example 62 includes the three dimensional display apparatus of example61, including or excluding optional features. In this example, the oneor more post processing technique includes screen-space ray tracing.

Example 63 includes the three dimensional display apparatus of any ofexamples 49-62, including or excluding optional features. In thisexample, the apparatus includes means for providing the display imagesto the display.

Example 64 includes the three dimensional display apparatus of any ofexamples 49-63, including or excluding optional features. In thisexample, the apparatus includes display backlight means for steeringlight alternating between a left and right eye based on the tracked eyelocation information at a refresh rate that is higher than a humanperceivable refresh rate.

Example 65 is a machine readable medium including code, when executed,to cause a machine to perform the method or realize an apparatus of anypreceding example.

Example 66 is an apparatus including means to perform a method as in anypreceding example.

Example 67 is machine-readable storage including machine-readableinstructions, when executed, to implement a method or realize anapparatus as in any preceding example.

Example 68 is a three dimensional display system including a processorand a display apparatus as in any preceding example.

Although example embodiments of the disclosed subject matter aredescribed with reference to circuit diagrams, flow diagrams, blockdiagrams etc. in the drawings, persons of ordinary skill in the art willreadily appreciate that many other ways of implementing the disclosedsubject matter may alternatively be used. For example, the arrangementsof the elements in the diagrams, and/or the order of execution of theblocks in the diagrams may be changed, and/or some of the circuitelements in circuit diagrams, and blocks in block/flow diagramsdescribed may be changed, eliminated, or combined. Any elements asillustrated and/or described may be changed, eliminated, or combined.

In the preceding description, various aspects of the disclosed subjectmatter have been described. For purposes of explanation, specificnumbers, systems and configurations were set forth in order to provide athorough understanding of the subject matter. However, it is apparent toone skilled in the art having the benefit of this disclosure that thesubject matter may be practiced without the specific details. In otherinstances, well-known features, components, or modules were omitted,simplified, combined, or split in order not to obscure the disclosedsubject matter.

Various embodiments of the disclosed subject matter may be implementedin hardware, firmware, software, or combination thereof, and may bedescribed by reference to or in conjunction with program code, such asinstructions, functions, procedures, data structures, logic, applicationprograms, design representations or formats for simulation, emulation,and fabrication of a design, which when accessed by a machine results inthe machine performing tasks, defining abstract data types or low-levelhardware contexts, or producing a result.

Program code may represent hardware using a hardware descriptionlanguage or another functional description language which essentiallyprovides a model of how designed hardware is expected to perform.Program code may be assembly or machine language or hardware-definitionlanguages, or data that may be compiled and/or interpreted. Furthermore,it is common in the art to speak of software, in one form or another astaking an action or causing a result. Such expressions are merely ashorthand way of stating execution of program code by a processingsystem which causes a processor to perform an action or produce aresult.

Program code may be stored in, for example, one or more volatile and/ornon-volatile memory devices, such as storage devices and/or anassociated machine readable or machine accessible medium includingsolid-state memory, hard-drives, floppy-disks, optical storage, tapes,flash memory, memory sticks, digital video disks, digital versatilediscs (DVDs), etc., as well as more exotic mediums such asmachine-accessible biological state preserving storage. Amachine-readable medium may include any tangible mechanism for storing,transmitting, or receiving information in a form readable by a machine,such as antennas, optical fibers, communication interfaces, etc. Programcode may be transmitted in the form of packets, serial data, paralleldata, etc., and may be used in a compressed or encrypted format.

Program code may be implemented in programs executing on programmablemachines such as mobile or stationary computers, personal digitalassistants, set top boxes, cellular telephones and pagers, and otherelectronic devices, each including a processor, volatile and/ornon-volatile memory readable by the processor, at least one input deviceand/or one or more output devices. Program code may be applied to thedata entered using the input device to perform the described embodimentsand to generate output information. The output information may beapplied to one or more output devices. One of ordinary skill in the artmay appreciate that embodiments of the disclosed subject matter can bepracticed with various computer system configurations, includingmultiprocessor or multiple-core processor systems, minicomputers,mainframe computers, as well as pervasive or miniature computers orprocessors that may be embedded into virtually any device. Embodimentsof the disclosed subject matter can also be practiced in distributedcomputing environments where tasks may be performed by remote processingdevices that are linked through a communications network.

Although operations may be described as a sequential process, some ofthe operations may in fact be performed in parallel, concurrently,and/or in a distributed environment, and with program code storedlocally and/or remotely for access by single or multi-processormachines. In addition, in some embodiments the order of operations maybe rearranged without departing from the spirit of the disclosed subjectmatter. Program code may be used by or in conjunction with embeddedcontrollers.

While the disclosed subject matter has been described with reference toillustrative embodiments, this description is not intended to beconstrued in a limiting sense. Various modifications of the illustrativeembodiments, as well as other embodiments of the subject matter, whichare apparent to persons skilled in the art to which the disclosedsubject matter pertains are deemed to lie within the scope of thedisclosed subject matter. For example, in each illustrated embodimentand each described embodiment, it is to be understood that the diagramsof the figures and the description herein is not intended to indicatethat the illustrated or described devices include all of the componentsshown in a particular figure or described in reference to a particularfigure. In addition, each element may be implemented with logic, whereinthe logic, as referred to herein, can include any suitable hardware(e.g., a processor, among others), software (e.g., an application, amongothers), firmware, or any suitable combination of hardware, software,and firmware, for example.

1-25. (canceled)
 26. A three dimensional display apparatus comprising: adisplay; a micro lens array; an eye tracker to track a plurality ofeyes, and to provide eye location information corresponding to thetracking; a rendering processor to render or capture color plus depthimages or light field images; and a light field processor to use the eyelocation information to convert the rendered or captured color plusdepth images or light field images to display images to be provided tothe display, and to use the eye location information to calculate an eyebox size.
 27. The three dimensional display apparatus of claim 26, therendering processor to render light field images based on the eyelocation information, and the light field processor to synthesizerendered or captured color, light field or multi view images to therequired light field image for the display.
 28. The three dimensionaldisplay apparatus of claim 26, comprising a display backlight to steerbacklight based on the eye location information.
 29. The threedimensional display apparatus of claim 26, wherein the display is a highpixel density display having a pixel density higher than 250 pixels perinch.
 30. The three dimensional display apparatus of claim 26, whereinthe micro lens array is a long focal length micro lens array with an Fnumber larger than
 8. 31. The three dimensional display apparatus ofclaim 26, wherein the eye tracker is a pupil tracker to track one ormore pupils, and wherein the eye location information is pupil locationinformation.
 32. The three dimensional display apparatus of claim 26,wherein the eye tracker is a three dimensional eye tracker and the eyelocation information is three dimensional eye location information. 33.The three dimensional display apparatus of claim 26, wherein therendering processor comprises a graphics engine.
 34. The threedimensional display apparatus of claim 26, wherein the light fieldprocessor is to convert the rendered or captured color plus depth imagesor light field images to stereo integral images to be provided to thedisplay.
 35. The three dimensional display of apparatus 26, wherein therendering processor is to render each color plus depth image or lightfield image according to the tracked location of one or more of theeyes.
 36. The three dimensional display apparatus of claim 26, whereinthe rendering processor is to render the color plus depth images orlight field images based on the eye location information.
 37. The threedimensional display apparatus of claim 26, wherein the light fieldprocessor is to use the eye location information to calculate an optimaleye box size and to offset displayed images to align a center of an eyebox with eye location of one or more of the plurality of eyes.
 38. Thethree dimensional display apparatus of claim 26, wherein the light fieldprocessor is to convert the rendered or captured color plus depth imagesor light field images to display images using one or morepost-processing technique.
 39. The three dimensional display apparatusof claim 38, wherein the one or more post processing technique includesscreen-space ray tracing.
 40. The three dimensional display apparatus ofclaim 26, wherein the light field processor is to provide the displayimages to the display.
 41. The three dimensional display apparatus ofclaim 26, comprising a display backlight to steer light alternatingbetween a left and right eye based on the tracked eye locationinformation at a refresh rate that is higher than a human perceivablerefresh rate.
 42. The three dimensional display apparatus of claim 26,wherein a spacing between the display and the micro lens array is set toa focal length of the micro lens array.
 43. The three dimensionaldisplay apparatus of claim 26, the light field processor to use the eyelocation information to minimize the eye box size.
 44. The threedimensional display apparatus of claim 26, the light field processor touse the eye location information to minimize the eye box size tocorrespond with a size of a pupil of one or more of the plurality ofeyes.
 45. The three dimensional display apparatus of claim 26, the lightfield processor to calculate the eye box size based on one or more of: adistance between the micro lens array and an eye viewpoint of one ofmore of the plurality of eyes, an integral image pitch size, a pixelpitch of the display, and a focal length of the micro lens array.
 46. Athree dimensional display light field image method comprising: trackingone or more eyes viewing a micro lens array and a display screen toprovide eye location information; rendering or capturing color plusdepth images or light field images; using the eye location informationto convert the rendered or captured color plus depth images or lightfield images to display images to be provided to the display; and usingthe eye location information to calculate an eye box size.
 47. The threedimensional display light field image method of claim 46, comprisingsteering light alternating between a left and right eye based on thetracked eye location information at a refresh rate that is higher than ahuman perceivable refresh rate.
 48. The three dimensional display lightfield image method of claim 46, comprising setting a spacing between thedisplay and the micro lens array to a focal length of the micro lensarray.
 49. The three dimensional display light field image method ofclaim 46, comprising minimizing the eye box size using the eye locationinformation.
 50. One or more tangible, non-transitory machine readablemedia comprising a plurality of instructions that, in response to beingexecuted on at least one processor, cause the at least one processor to:track one or more eyes viewing a micro lens array and a display screento provide eye location information; render or capture color plus depthimages or light field images; use the eye location information toconvert the rendered or captured color plus depth images or light fieldimages to display images to be provided to the display; and use the eyelocation information to calculate an eye box size.
 51. The one or moretangible, non-transitory machine readable media of claim 50, comprisinga plurality of instructions that, in response to being executed on atleast one processor, cause the at least one processor to: render lightfield images based on the eye location information; and synthesizecaptured color, light field or multi view images to the required lightfield image for the display.
 52. The one or more tangible,non-transitory machine readable media of claim 50, comprising aplurality of instructions that, in response to being executed on atleast one processor, cause the at least one processor to track one ormore pupils viewing the micro lens array and the display screen toprovide pupil location information, wherein the eye location informationis pupil location information.
 53. The one or more tangible,non-transitory machine readable media of claim 50, comprising aplurality of instructions that, in response to being executed on atleast one processor, cause the at least one processor to render thecolor plus depth images based on the eye location information.
 54. Theone or more tangible, non-transitory machine readable media of claim 50,comprising a plurality of instructions that, in response to beingexecuted on at least one processor, cause the at least one processor tosteer light alternating between a left and right eye based on thetracked eye location information at a refresh rate that is higher than ahuman perceivable refresh rate.
 55. The one or more tangible,non-transitory machine readable media of claim 50 comprising a pluralityof instructions that, in response to being executed on at least oneprocessor, cause the at least one processor to minimize the eye box sizeusing the eye location information.
 56. The one or more tangible,non-transitory machine readable media of claim 50, comprising aplurality of instructions that, in response to being executed on atleast one processor, cause the at least one processor to use the eyelocation information to minimize the eye box size to correspond with asize of a pupil of one or more of the eyes.
 57. The one or moretangible, non-transitory machine readable media of claim 50, comprisinga plurality of instructions that, in response to being executed on atleast one processor, cause the at least one processor to calculate aneye box size based on one or more of: a distance between the micro lensarray and an eye viewpoint of one of more of the plurality of eyes, anintegral image pitch size, a pixel pitch of the display, and a focallength of the micro lens array.
 58. A three dimensional displayapparatus comprising: means for tracking one or more eyes viewing amicro lens array and a display screen to provide eye locationinformation; means for rendering or capturing color plus depth images orlight field images; means for using the eye location information toconvert the rendered or captured color plus depth images or light fieldimages to display images to be provided to the display; and means forusing the eye location information to calculate an eye box size.
 59. Thethree dimensional display apparatus of claim 58, comprising: means forrendering light field images based on the eye location information; andmeans for synthesizing captured color, light field or multi view imagesto the required light field image for the display.
 60. The threedimensional display apparatus of claim 58, comprising means for steeringlight alternating between a left and right eye based on the tracked eyelocation information at a refresh rate that is higher than a humanperceivable refresh rate.
 61. The three dimensional display apparatus ofclaim 58, comprising means for minimizing the eye box size using the eyelocation information.